Gartner: IT staff need new skills to manage big data

Bill Goodwin is Computer Weekly's management editor.

He is an award winning journalist, who has written for national newspapers, magazines and has worked as a researcher on investigative television documentaries. He developed and taught an investigative journalism course at the University of Westminster, where he is currently an external examiner. His awards include a Freedom of Information Award, and the British Editors Freedom of the Media Award. Most recently he won three awards in the BT Security Journalism Awards 2007, including the BT ‘Enigma Award’ in recognition of his outstanding and consistent contribution to IT security journalism.

See all articles by Bill Goodwin >>

Follow Bill on Twitter

Linked-In profile for Bill Goodwin

[email protected] 020 8652 8644 Active Bill Goodwin False True

Enterprises will need to re-imagine their IT department if they are to successfully manage the explosive growth of data within their organisation, claims analyst group Gartner.

CIOs are under increasing pressure to harness the "big data" within their organisations, as companies look for ways to understand their customers and predict changing demand more effectively.

And those that succeed stand to outperform their industry peers by 20%, the analyst group claims.

But IT departments are still a long way from having the right skills to make analysing big data a reality, says Debra Logan vice-president and distinguished analyst at Gartner.

"You need to have people who can think about information needs, and that is a different skillset to IT. It's a profound change that everybody needs to get to grips with," she said in an interview with Computer Weekly.

Sign up to Computer Weekly to download the latest in-depth articles from Gartner analysts:

• The reality behind government cloud computing hype
• Software asset management standards and best practice adoption

CIOs are going to have to look outside of the traditional IT department to find data scientists and information management specialists who can help them to make sense of corporate data.

The role of IT professionals will shift from managing IT to managing information, as big data accelerates the move towards the cloud.

"I know we say all the time that IT is changing beyond recognition, but it is. Internal IT is becoming irrelevant. In five to 10 years you will be able to get all your services outside your company," she said.

The biggest challenge for CIOs will be to work out how to make the increasing volumes of data organisations have to deal with manageable, she says.

She advises businesses to bring staff from different parts of the business together to understand what information is genuinely important, and what - the majority - can be discarded.

The developing world of data analytics

Supermarket group Walmart is one example of a company that has mastered analysing big data in its supply chain. Similarly Tesco has created a sophisticated data mining operation for its store card data.

But neither firm has yet begun tackling the analysis of unstructured data.

"We are going to see a surge in analytics technologies," said Logan." Business intelligence has been used to analyse structured data sets, but they only work for structured data. Now people are wanting to bring in hybrid contents analytics - the question is, how do you analyse structured and unstructured data together?"

Solving this problem will gave more accurate ways of assessing customer satisfaction and predicting changing demand for products much further in advance.

However technology is still at an early stage. Major suppliers such as HP and EMC are focused more on selling large boxes than on developing new big data products, says Logan. And the future of Autonomy, a market leader in data analytics, is not yet clear following its acquisition by HP.

Apache Hadoop, open source software that supports distributed analysis of petabytes of data, has attracted the interests of suppliers, but is not yet on the radar of most users, she says.

Early adopters include daily deal site Groupon, which uses the technology to analyse transaction data generated by over 70 million registered users worldwide, and New York Stock Exchange operator NYSE Euronext, which uses the technology to manage petabytes of transaction data.

US bookseller Barnes & Noble is using a version of Hadoop to understand customer preferences and buying patterns across three sales channels: its retail outlets, its online store and e-reader downloads.

Debra Logan is chair of the Gartner Symposium in Barcelona (7-10 November 2011).

Sign up to Computer Weekly to download more in-depth articles from Gartner analysts:

• Building datacentres using containers and modules
• How to transform legacy code into object-orientated languages
• Why you need an enterprise strategy for social business
• The future of client computing